Merged from ndsep_phase44_final.tar and ndsep_phase44_final_20260426_181302.tar.
Uses the latest (April 26) tarball as the base with all Phase 35-44 changes.

Includes:
- Full-stack TypeScript app (React client + Node.js/Express server)
- PostgreSQL/Drizzle ORM database layer
- Worker services (Go, Python, Rust)
- Infrastructure configs (Docker, K8s, Airflow, Prometheus)
- Mobile apps (Flutter, React Native)
- E2E tests (Playwright)
- CI/CD workflows
- Security audit reports and compliance tooling

Cleaned up build artifacts (compiled binaries, Rust target, __pycache__)
and updated .gitignore accordingly.

Co-Authored-By: Patrick Munis <pmunis@gmail.com>

…from DB

Previously requireSession used req.cookies which requires cookie-parser middleware.
Now extracts token from raw Cookie header directly (using 'cookie' package) and
looks up the full user object from the database (including role) for proper
admin authorization checks.

Co-Authored-By: Patrick Munis <pmunis@gmail.com>

…ard & Middleware Health routes

- Moved catch-all NotFound route from middle of Switch to the end, unblocking
  13+ routes (data-pipeline, data-lineage, knowledge-graph, penalty-dashboard, etc.)
- Added SecurityDashboard and MiddlewareHealth imports and routes
- Removed duplicate /dpco route (DpcoLanding vs DpcoPortal)
- Added /security-dashboard and /middleware-health sidebar entries
- All 22 compliance module routes now render correctly (0 remaining 404s)

Co-Authored-By: Patrick Munis <pmunis@gmail.com>

…stency

- Reorganize sidebar from flat menuItems array to 10 functional category groups:
  Core Platform, Enforcement & Finance, Compliance Management, DPCO Portal,
  Organizations & IAM, AI & Intelligence, Operations & Infrastructure,
  Banking & Sectors, Governance & Reporting, Advanced Features, Admin & Settings
- Add collapsible section headers with color-coded badges and item counts
- Fix DPCO page SelectItem empty value error (use 'all' instead of '')
- Replace hardcoded dark theme classes with theme-aware Tailwind utilities
- Use Card/CardContent/CardHeader/CardTitle components for consistent UI
- Replace raw HTML select with Select/SelectContent/SelectItem components
- Replace raw div progress bars with Progress component

Co-Authored-By: Patrick Munis <pmunis@gmail.com>

… data display

- enforcement_fines: org_id → organization_id, remove case_id join
- vendor_risk: contract_status → status in stats query
- compliance_gap: assessed_at → created_at
- regulatory_intelligence: published_at → created_at
- whistleblower: submitted_at → created_at
- incident_response: incident_type → category, activated_at → created_at
- data_pipeline: fix dbt_models schema→schema_name, remove is_paused, dag_name→dag_id
- ai_ethics: overall_ethics_score → overall_score, review_status → status
- cross_agency: status 'active' → 'approved' in stats
- staff_training (db.ts): training_status → training_type, scheduled_date → created_at
- enforcement_timeline (newFeatures.ts): cv.violation_type → cv.title

Co-Authored-By: Patrick Munis <pmunis@gmail.com>

… fixes

- Created 10 banking tables (banking_institutions, kyc_records, aml_cases,
  watchlist_entries, nip_transactions, rtgs_transactions, swift_messages,
  fraud_alerts, cbn_reports, correspondent_banks)
- Seeded all 98 tables with 830 total rows of realistic Nigerian data
- Fixed banking router: MySQL ? placeholders → PostgreSQL $N params
- Fixed banking router: LIKE → ILIKE for case-insensitive search
- Added scripts/seed-all.sql — standalone SQL seed file
- Added scripts/seed-comprehensive.mjs — Node.js wrapper with verification
- Added npm scripts: seed:all, seed:all:force
- Updated banking router connection string to match .env credentials
- Zero empty tables across the entire platform

Co-Authored-By: Patrick Munis <pmunis@gmail.com>

…tor, uptime, dashboard

Components:
- Rust SNMP/Syslog/NetFlow collector (port 8190) with 23 OID mappings, simulation mode
- Go escalation engine (port 8191) with 5 policies, 2 on-call schedules, 4 runbooks
- Python cross-domain alert correlator (port 8192) with temporal/causal/topological/statistical strategies
- Rust uptime tracker (port 8193) monitoring 27 services with p50/p95/p99 response times
- TypeScript tRPC NOC router with 30+ unified procedures aggregating all subsystems
- React NOC Dashboard with 7 tabs: Overview, Alerts, Topology, Uptime/SLA, Escalation, Collectors, Correlation
- PostgreSQL migration with 10 NOC tables and 23 indexes
- K8s manifests for all 4 NOC microservices
- Middleware integration: Kafka, Redis, OpenSearch, Dapr, Fluvio, Lakehouse, Temporal, APISIX, TigerBeetle, Mojaloop, OpenAppSec, Keycloak, Permify

Co-Authored-By: Patrick Munis <pmunis@gmail.com>

…-to-PostgreSQL queries, add route aliases

- Add route aliases for /noc-dashboard, /liveness-verification, /wiredigg
- Fix phase6Features.ts: convert MySQL ? placeholders to PostgreSQL $1 syntax
- Fix phase6Features.ts: convert ON DUPLICATE KEY UPDATE to ON CONFLICT DO UPDATE
- Fix phase6Features.ts: use pool.query() instead of drizzle execute for raw SQL
- Add migration: 13 missing columns across 8 tables (ndpa_index, training_status,
  privacy_notice_status, dpa_status, dpo_report_status, adequacy_status,
  significant_effect, parental_consent_status, transfer_instrument_status,
  export_job_status)
- Add migration: 3 missing tables (onboarding_checklists, changelogs,
  compliance_audit_returns)
- TypeScript compiles clean (0 errors)
- All 95 sidebar routes return HTTP 200

Co-Authored-By: Patrick Munis <pmunis@gmail.com>

Tier 1 (Production-Ready):
- PostgreSQL persistence: 10 new tables (dt_jurisdictions, dt_policies,
  dt_sector_models, dt_org_agents, dt_simulations, dt_simulation_results,
  dt_monte_carlo_stats, dt_policy_impacts, dt_economic_indicators,
  dt_sandboxes, dt_bilateral_agreements) with 32 indexes
- Go service rewrite: 1,400+ lines replacing 365-line prototype
- DB-backed state: jurisdictions, sectors, policies all from PostgreSQL
- Monte Carlo in Go: 100+ iterations with confidence intervals (p5/p25/p50/p75/p95)
- Kafka/Dapr event publishing structure for simulation events

Tier 2 (Real Modeling):
- Rust Monte Carlo engine (:8177): 1000+ parallel iterations via Rayon,
  per-sector CI, timeline CI, GDP impact estimation
- Rust Agent-Based Model (:8178): per-org agents with budget, staff,
  tech maturity, risk appetite, peer pressure, network effects
- Rust System Dynamics (:8179): Forrester stock-and-flow model with
  10 stocks, 10 flows, 4 causal feedback loops, sensitivity analysis
- Python ML predictor (:8176): XGBoost-style breach prediction for
  30 orgs across 4 jurisdictions, economic impact modeling, network
  effects propagation analysis

Tier 3 (Multi-Government Policy Engine):
- 8 jurisdictions: NG, GH, KE, ZA, EU, RW, SN, TZ
- Policy Definition Language: JSONB rules + parameters
- Policy composition with conflict detection
- Counterfactual analysis (baseline vs hypothetical)
- Regulatory sandbox (isolated policy testing)
- Economic indicators + bilateral agreements

Integration:
- 30+ new tRPC procedures (Monte Carlo, ABM, System Dynamics,
  ML prediction, policy compose, counterfactual, sandbox, economics)
- Enhanced React UI: 8 tabs (Ecosystem, Simulator, Predictions,
  Policies, Counterfactual, Sandbox, Economics, History)
- K8s manifests for all 4 new microservices
- TypeScript compiles clean (0 errors)

Co-Authored-By: Patrick Munis <pmunis@gmail.com>

…middleware health, seed data scaling

- Add production seed data migration (000019): orgs 28→106, breaches 13→215,
  alerts 13→103, audit logs 175→480, ML predictions 12→155, consent 20→233
- Add error monitoring module with sliding window, alert thresholds, Sentry integration
- Add Keycloak OIDC authentication (JWT validation, role mapping, graceful fallback)
- Add middleware connection manager with real HTTP health probes for all 14 services
- Add circuit breakers for all external service connections
- Add worker binary builder (auto-compile Go/Rust binaries before starting)
- Add productionReadiness tRPC router (error summary, middleware health, auth status,
  readiness score, seed data summary)
- Wire error monitoring into uncaughtException/unhandledRejection handlers
- Add /api/errors/summary and /api/middleware/health Express endpoints
- Start background health monitor on server boot
- Add 12 production indexes for high-traffic query optimization
- TypeScript compiles clean (0 errors)

Co-Authored-By: Patrick Munis <pmunis@gmail.com>

…on engines to Go orchestrator

- Install Ollama v0.24.0 with llama.cpp backend, pull qwen2.5:1.5b model
- Update ollama_llm_worker.py: Qwen first in model preference (qwen2.5 > mistral > llama3)
- Update noc_agent_reasoning.py: default model changed to qwen2.5:1.5b
- Update ai_compliance_engine.py: default model changed to qwen2.5:1.5b
- Add llama.cpp native inference worker (port 8204) as Ollama fallback
- Add llama.cpp fallback chain in ollama_llm_worker generate()
- Wire 3 Rust simulation engines into Go Digital Twin orchestrator:
  - Monte Carlo (port 8177): Rayon-parallelized stochastic CI
  - Agent-Based Model (port 8178): per-org peer pressure simulation
  - System Dynamics (port 8179): Forrester stock-and-flow causal loops
- Add circuit breaker pattern for Rust service health checks
- Graceful degradation: Go linear model fallback when Rust unavailable
- Health endpoint reports Rust engine availability status
- Add scripts/install-ollama.sh for automated setup
- All compilers pass: Go, Rust (3 crates), TypeScript (0 errors)

Co-Authored-By: Patrick Munis <pmunis@gmail.com>

…STM/SHAP), GNN compliance engine (GraphSAGE/link prediction)

LAKEHOUSE:
- New lakehouse_analytics_engine.py: DuckDB + Parquet-based analytics
- ETL pipeline: PostgreSQL → Parquet (7 tables, partitioned)
- 6 materialized views (sector compliance, breach trend, penalty analytics, etc.)
- Feature serving for ML model training
- Time-travel snapshots, compaction, SQL query API
- Rust lakehouse_ingest now forwards records to analytics engine
- MinIO + Iceberg setup script (scripts/setup-lakehouse.sh)

ML/DL:
- New ml_production_engine.py with 4 real trained models:
  - XGBoost breach predictor (trained on breach_incidents + orgs)
  - LSTM-style violation forecaster (6-month ahead predictions)
  - IsolationForest anomaly detector (200 estimators)
  - RandomForest multi-class risk scorer (4 risk tiers)
- SHAP TreeExplainer for XGBoost (feature-level explanations)
- Auto-retraining scheduler (configurable interval)
- Model versioning + artifact persistence via joblib

GNN:
- New gnn_compliance_engine.py:
  - Builds compliance graph from PostgreSQL (orgs, violations, enforcement, breaches)
  - GraphSAGE 3-layer message passing with learned weight matrices
  - 32-dim node embeddings with ReLU + L2 normalization
  - Link prediction (LogisticRegression on concatenated GNN embeddings)
  - Future violation prediction per org
  - Graph path finding, node similarity, neighbor queries

INTEGRATION:
- 3 new tRPC routers: lakehouseAnalytics, mlProduction, gnn
- 9 new Express REST endpoints (/api/lakehouse/*, /api/ml/*, /api/gnn/*)
- 3 new worker definitions in workerManager.ts
- AI health dashboard expanded to 10 services (was 7)
- All TypeScript compiles clean (0 errors)
- All Rust crates compile clean
- All Go builds pass

Co-Authored-By: Patrick Munis <pmunis@gmail.com>

- Lakehouse: Fix 6 table queries (risk_level→risk_score, status→compliance_status, etc.)
- ML: Fix risk_level→risk_score, status→compliance_status filter
- GNN: Fix organizations/violations/breach SQL column references
- All services now successfully query real PostgreSQL data

Co-Authored-By: Patrick Munis <pmunis@gmail.com>

… Lakehouse integration

- GraphSAGE GNN: 3-layer PyTorch nn.Module with LEARNED weights via BCELoss + Adam
  backpropagation, link prediction MLP, 9,441 trainable parameters, test_accuracy=0.88
- LSTM Forecaster: PyTorch nn.LSTM (2-layer, hidden_dim=64) with BPTT training on
  time-series violation data, 53,313 parameters, saves .pt checkpoint files
- Autoencoder Anomaly Detection: PyTorch encoder-decoder with latent_dim=16, replaces
  broken IsolationForest, 1,819 parameters, reconstruction-error-based thresholding
- XGBoost + SHAP: Real trained XGBoost with TreeExplainer, cross-validation (cv=0.99)
- Ray 2.55.1: Distributed training support (train all 4 models in parallel via Ray)
- Lakehouse: DuckDB reads PostgreSQL → Parquet ETL, materialized sector views
- MLOps: Experiment tracker with versioned artifacts, model registry with 5 entries
- Express proxy routes: 10 new /api/ray-ml/* endpoints on main app
- Worker manager: ray-ml-engine registered on port 8250
- All models 100% CPU-native (PyTorch CPU, no CUDA dependency)

Co-Authored-By: Patrick Munis <pmunis@gmail.com>

…nger, feedback loop, warm-start

Added LAYER 7: Continuous Training Pipeline to Ray ML Engine (v5.0.0):

Data Drift Detection:
- KS-test (scipy.stats.ks_2samp) and PSI per feature
- Configurable thresholds via env vars (DRIFT_THRESHOLD_KS, DRIFT_THRESHOLD_PSI)
- Automatic drift history tracking (last 100 checks)
- Baseline auto-set from training data

Scheduled Auto-Retraining:
- Background thread with configurable interval (RETRAIN_INTERVAL, default 6h)
- Drift-triggered retraining when feature distributions shift
- Manual trigger via POST /continuous/trigger
- Start/stop via POST /continuous/start and /continuous/stop

Incremental/Warm-Start Learning:
- LSTM and Autoencoder load last checkpoint before training
- Warm-started models use lower learning rate (0.0005 vs 0.001)
- Fewer epochs when warm-starting (80/60 vs 200/150)
- Latest checkpoint saved alongside versioned weights

Prediction Feedback Loop:
- All predictions auto-logged to JSONL feedback store
- POST /feedback/ingest to record actual outcomes
- Feedback pairs available per model for retraining
- Stats endpoint shows prediction/feedback counts per model

Champion/Challenger Model Promotion:
- New model versions compared against current champion
- Promote only if improvement exceeds threshold (default 1%)
- Full promotion history with before/after scores
- Auto-promote on first training (no existing champion)

Lakehouse Auto-Sync:
- ETL refresh (PostgreSQL → Parquet) runs before each retraining
- Ensures models always train on latest data

Retraining Event Log:
- Every retrain logged with trigger type, duration, before/after metrics
- Persisted to disk as JSON files
- Stats endpoint shows trigger distribution and avg duration

Express Proxy Routes (11 new endpoints):
- /api/ray-ml/continuous/{start,stop,status,trigger,config}
- /api/ray-ml/drift/{report,history}
- /api/ray-ml/feedback/{ingest,stats}
- /api/ray-ml/champion/info
- /api/ray-ml/retrain/{events,status}

Environment Variables:
- CONTINUOUS_TRAINING_ENABLED, RETRAIN_INTERVAL, DRIFT_CHECK_INTERVAL
- DRIFT_THRESHOLD_KS, DRIFT_THRESHOLD_PSI, CHAMPION_THRESHOLD

Co-Authored-By: Patrick Munis <pmunis@gmail.com>

…ng, GNN/ML lakehouse features

- Fix orchestration journeys port mismatch (8210 → 8140) — all 12+ journey lakehouse calls now reach the analytics engine
- Implement incremental ETL: uses WHERE incremental_col > last_sync for delta extraction instead of full re-extract
- Add data lineage tracking: every ETL run records source, destination, row counts, timing
- Make Rust NOC collector publish_to_lakehouse() real: POST /ingest to analytics engine (was log::debug stub)
- Make Python NOC correlator publish_to_lakehouse() real: POST /ingest with retry (was log.debug stub)
- Fix Rust lakehouse_writer: forwards features + predictions to Lakehouse Analytics Engine for Parquet offline store (was PostgreSQL-only)
- Connect GNN engine to Lakehouse: tries Lakehouse compliance_features first, falls back to PostgreSQL; publishes embeddings back to Lakehouse after graph build
- Connect ML Production Engine to Lakehouse: tries Lakehouse features first for training data, falls back to direct PostgreSQL
- Add 4 new Express proxy endpoints: /api/lakehouse/lineage, /api/lakehouse/incremental/status, /api/lakehouse/etl/reset, /api/lakehouse/snapshots
- Add 4 new tRPC procedures: lineage, incrementalStatus, resetIncremental, ingest
- Add reqwest dependency to lakehouse_writer Cargo.toml for HTTP forwarding

Co-Authored-By: Patrick Munis <pmunis@gmail.com>